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Abstract Body 



Context: 

The U.S. began a new national standards movement in the area of K-12 scienee education 
curriculum reform in the 1980s known as “Science for All” to develop a population that is 
literate in economic and democratic agendas for a global market focused on science, technology, 
engineering, and mathematics (STEM) (Duschl, 2008). The National Research Council (NRC) 
report. Taking Science to Schooi: Learning and Teaching Science in Grades /f-<9(TSTS; NRC, 
2007b) described shortages in attracting students to science learning and careers, and of science 
teachers (particularly women and minorities). More recently, researchers have focused on 
science reform that incorporates a cultural imperative in the teaching of science (Driver, Leach, 
Millar, & Scott, 1996; Millar, 2006; Osborne, Duschl, & Fairbrother, 2002). The sister NRC 
report. Rising above the Gathering Storm (RAGS, NRC, 2007a), describes four areas of needed 
proficiency for science students of how to: generate and evaluate scientific evidence and 
explanations; know, use, and interpret scientific explanations of the natural world; understand the 
nature and development of scientific knowledge; and participate productively in scientific 
practices and discourse. To this end, pedagogical skills in science education have moved from 
teaching students how to memorize what they need to know from science textbooks to 
developing an understanding of the knowledge-building process by learning how to develop 
explanations and predictions about our world. 

The two NRC reports demonstrate changes in pedagogy and instruction that appear to be 
better suited to the evolving technological world. As members of society are expected to process 
information that is updated constantly and rapidly, it is critical to understand how ideas are 
developed and processed. Research in abstract reasoning teaches us that infants learn causal 
inference and differentiation of animate and inanimate objects, demonstrating that the learning 
ability of even the youngest children permits them to engage in complex decision making 
(Gelman & Brenneman, 2004; Mertz, 2004; Spelke, 2000). To do this, students require abstract 
deductive and inductive reasoning skills, including the ability to view with an open mind and a 
willingness to be aware of the world (Critical Thinking Co., 201 1). 

Measuring student success. Raudenbush (2008) argues that, in contrast to past models 
that describe conventional resources such as per pupil expenditures, teacher credentials, physical 
facilities, or class size (Cohen, Raudenbush, & Ball, 2003) as the direct cause for student 
outcomes, instruction is the proximal cause for student learning and thereby places the emphasis 
on the continuous classroom interplay of assessment and instruction. One such current 
pedagogical practice is the Science Writing Heuristic (SWH) approach, which combines current 
understandings of learning as a cognitive and negotiated process with the techniques of 
argument-based inquiry (Duschl & Grandy, 2007; Hand, 2007), critical thinking skills, and 
writing to strengthen student outcomes. This enables students to develop critical thinking, habits 
of mind, and communications, and these abilities result in cognitive and meta-cognitive attributes 
that foster understanding of the nature of science, scientific inquiry, and the big ideas of science. 

Fostering scientific discourse. The importance of the use of language in science has 
become prominent in the science education literature (e.g., Dawes, 2004; Shelley, Yore, & Hand, 
2009; Yore, Bisanz, & Hand, 2003). Argumentation, known as “the language of science” 
(Duschl, Ellenbogan, & Eurduran, 1999), is used currently by researchers to promote classroom 
learning. Students are able to construct meaning as they interpret and reinterpret events through 
the argumentation-driven lens of their prior knowledge (Berk & Winsler, 1995; Tippett, 2009). 
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The SWH approach. The SWH approach to learning is based on improving students’ 
understanding of science by embedding science argument within typical inquiry lessons. Within 
this approach teachers are required to align their pedagogical practices with how children learn. 
Students are required to pose questions, generate claims and evidence, compare their answers to 
others, and reflect on changes in their understanding. Several studies have suggested that this 
type of inquiry approach promotes critical thinking and reasoning, as students are required to use 
oral and written language to negotiate their understanding of science. For example, Akkus, 
Gunel, and Hand (2007) demonstrated significant gains in student performance on science 
components of standardized tests such as the Iowa Tests of Basic Skills (ITBS) and the Iowa 
Tests of Educational Development (ITED) comparing student test performance between 
classrooms with high levels of traditional science teaching and those with high-quality SWH 
implementation. The effect size difference of 1.23 between high and low student achievers and 
only 0.13 for students in classrooms where teachers used high levels of SWH teaching indicates 
that the SWH approach is effective for all learners in the classroom. 

Other studies of applications of SWH pedagogical teaching in chemistry (Anderson & 
Bodner, 2008; Bhattacharyya & Bodner, 2005; Rudd, Greenbowe, & Hand, 2007) compared to 
traditional formats, showed significant association between higher explanation scores and the 
SWH format. Hohenshell and Hand (2006) showed that 10th grade biology students scored 
significantly better than control students on conceptual questions after completing laboratory 
activities using traditional and SWH approaches (E(l,43)=5.53, /7=0.023, partial q^=0.114). 
Poock, Burke, Greenbowe, and Hand (2007) demonstrated benefits for female students from 
high-implementation use of the SWH approach. Gunel (2006) found larger effect size changes in 
students’ scores (1.0, vs. 0.4 or less) from higher levels of implementation of the SWH approach 
on students’ performance on ITBS/ITED science tests across a 3-year period. 

Two-sample /-tests conducted on 5th grade Cornell Critical Thinking (CCT) data 
(Shelley et ah, forthcoming; Villanueva et ah, 2011) demonstrated that SWH students (/7=1,154), 
compared to control students (/7=882) had significantly higher gains overall (/7=.002), as well as 
in levels of science induction (/7=.010) and deduction (/7=.004), but did not show significantly 
different gains on observation and credibility (p=.322) nor the assumptions underlying science 
deduction (/7=.191). A structural equation model (SEM) (Shelley et ah, forthcoming) also showed 
that ITBS achievement measures of Reading Comprehension (RC), Math Concepts/Estimation 
(Ml), Math Problems and Data Interpretation (M2), Science Comprehension (NSI), and Science 
Inquiry (SI) form a single overall metric of student outcomes, and that each score is related 
significantly to overall student achievement. Hand et al. (2011) showed that the SWH treatment 
had a significantly positive effect in SEM results on student gains from pretest to posttest (IMP) 
on CCT test scores (see Eigure I; SWH=I, Control=0). 

Purpose: 

The purpose of this paper is to examine the impact of implementation of the SWH 
approach at 5th grade level in the public school system in Iowa as measured by CTT student test 
(Ennis & Millman, 2005) scores. This is part of a project that overall tests the efficacy of the 
SWH inquiry-based approach to build students’ content knowledge, argumentation skills, and 
interest in science to construct the foundation of science literacy with elementary school 
children, so that all students “become familiar with modes of scientific inquiry, rules of 
evidence, ways of formulating questions and ways of proposing explanations” (National 
Research Council [NRC], 1996, p. 21). 
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Setting: 

A description of the SWH study by letter, followed by an in-person meeting, was 
completed in the summer of 2009 with school district superintendents in Iowa to obtain 
permission for participation by elementary school buildings in the study. After obtaining consent 
from the district superintendents, a total of 48 schools were recruited into the study. 

Participants: 

The study was conducted on Iowa elementary school students, in grades 3-6, with 24 
school buildings randomly assigned to treatment and 24 to control. Data for ITBS were collected 
on over 60,000 students; CCT scores, measured only at 5th grade, were obtained on over 2,000 
students at pretest and posttest. 

Intervention: 

Teachers in school districts randomized to the intervention group were trained in the 
SWH technique during the summer of 2009 at workshops held at four geographic regions of 
Iowa. This training took part over three days and included specific training on the SWH 
approach including how to foster argumentation skills in students in the classroom. University of 
Iowa SWH staff obtained video recordings of individual teachers’ performance in science 
classroom at different times over the academic year. Classroom implementation has continued 
since Fall 2009; all selected schools remain in the study. 

Research Design: 

An experimental design was employed, with random assignment of participating 
elementary school buildings to SWH treatment or Control condition. Once recruitment of 
buildings was completed, blocks were formed for the purposes of randomization. Blocks were 
either districts with multiple buildings or districts that were similar in enrollment based on 
percentage of students on free and reduced lunch or certified enrollment. Two exceptions to this 
randomization strategy were as follows: (1) two religious schools of comparable size were 
blocked together, and the other religious school, of very small size, was paired with another 
school of very small size; and (2) 10 schools not randomized initially because their data arrived 
later were randomized into districts as we received them. 

Data Collection and Analysis: 

ITBS student scores were obtained for all students from the years immediately prior to 
the start of the SWH study. These ITBS scores included all composite scores and a subgroup 
score in science. The CCT test was administered in a Fall 2009 pretest and spring 2010 posttest; 
these results were combined with ITBS scores for data analyses. A multilevel model was 
estimated to assess the relative contributions of individual student (Level 1) variables, classroom 
(Level 2), and randomization unit (Level 3) which is a single building except where clustering of 
buildings was necessary owing to small enrollment. We treat student as nested within classroom 
and classroom as nested within randomization unit. 

Results: 
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The model predicting change in CCT scores between pretest and posttest was estimated 
using R software by a linear mixed model fit by restricted maximum likelihood (Table 1). The 
estimated model, using R notation, is: 

IMP ~ 1 + Trt + ASN + WHT + SED + GAT + ERL + TIM + TIE + PreTest + 

ASN:T1M + SED:T1E + Trt:TlE + Trt:SED + (1 I TID) + (1 I Unitl) 
where IMP is posttest CCT scores minus pretest CCT score, Trt is SWH treatment vs. control, 
ASN is Asian American, WHT is white non-Hispanic, SED is special education, GAT is gifted 
and talented, ERE is free and reduced lunch, TIM is Type 1 remedial mathematics, TIE is Type 

1 remedial language, PreTest is the pretest CCT test score, ASN:T1M is the interaction between 
Asian- American and Type 1 mathematics, SED:T1E is the interaction between special education 
and Type 1 language, TrtrTlE is the interaction between treatment/control and Type 1 language, 
Trt:SED is the interaction between treatment/control and special education, (1 I TID) is the Eevel 

2 effect of classroom, and (1 I Unitl) is the Eevel 3 effect of randomization unit. The model was 
estimated on 2,009 5th grade students who completed both the CCT pretest and posttest at Eevel 
1,81 classrooms at Eevel 2, and 47 randomization units at Eevel 3. The random effects results 
show that classroom (TID) accounts for only about 2% of explained variance, and randomization 
unit (Unitl) accounts for about 7%. Effects not in the model are Asian American (AI), Hawaiian 
or Native Alaskan (HAW), English language learner (EEE), Section G504 special education 
status (G504), and African American (BEK), as well as related interactions, which were not 
significant in the presence of the other effects in preliminary models. 

Eixed effect results for the model indicate significantly positive effects at Eevel 1 for 
SWH treatment compared to control (Trt), Asian-American student ethnicity (ASN), non- 
Hispanic white student ethnicity (WHT), gifted and talented status (GAT), and the interaction of 
Asian-American ethnicity with Type 1 Math (ASN:T1M); significantly negative Eevel 1 effects 
were estimated for special education status (SED), free and reduced lunch status (ERE), Type 1 
Math (TIM), pretest score (PreTest), and the interaction between treatment and Type 1 
Eanguage (Trt:TlE). The main effect of Type 1 Eanguage (TIE), the interaction between special 
education and Type 1 Eanguage (SED:T1E) and the interaction between treatment and special 
education (Trt: SED) were not significant. 

Conclusions: 

These multilevel results clearly support the conclusion of significant CCT improvements 
due to SWH implementation compared to control, after accounting for appropriate covariates. 
The relatively minimal variance components attributable to randomization unit and classroom 
support the validity of the randomization strategy and the lack of confounding attributable to 
innate differences in teacher characteristics. 

The results reported here are for a multilevel model that uses individual student 
differences between posttest and pretest CCT scores to measure “improvement.” Subsequent 
analysis will incorporate level of teacher implementation based on classroom observations, use 
maximum likelihood estimates for missing data replacement to enhance model power, and 
compare different missing data substitution methods (mean overall, student mean, question 
mean, a combination of student and question means). We will try using the above as probabilities 
in Bernoulli trails and bootstrap estimate to see how variance estimates change. We also will 
investigate subcategories of CCT and ITBS data, by addressing the nested data structure instead 
of treating the sum of scores across subcategories as approximately normal. 
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Figure 1. Preliminary Structural Equation Model 
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Table 1. Summary of Results of Multilevel Model for Pretest to Posttest Changes In Cornell 
Critical Thinking Test 



AIC=12773; BIC=12868; log likelihood=-6370; deviance= 12749; REMLdev=12739 



Random effects: 



Groups 


Name 


Variance 


Std. Dev. 


TID (Intercept) 


0.71943 


0.8482 


Unitl (Intercept) 


2.41478 


1.5540 


Residual 




32.29859 


5.6832 


Fixed effects: 






Estimate Std. 


Error 


/value 


(Intercept) 


16.0662 


1.0055 


15.978 


Trt 


1.5960 


0.5950 


2.683 


ASN 


1.5677 


0.9947 


1.576 


WHT 


1.7977 


0.6448 


2.788 


SED 


-3.2987 


0.7016 


-4.701 


GAT 


3.0565 


0.4165 


7.339 


ERE 


-0.8768 


0.2924 


-2.998 


TIM 


-2.9743 


0.9493 


-3.133 


TIE 


-0.3999 


0.9917 


-0.403 


PreTest 


-0.4094 


0.0193 


-2I.2II 


ASN:TIM 


16.9620 


5.8986 


2.876 


SED:TIE 


3.0970 


1.6634 


1.862 


Trt:TIE 


-2.3192 


1. 1409 


-2.033 


Trt: SED 


-I.66I5 


0.8755 


-1.898 



Analysis of Variance Table 





Df 


SumSq 


Mean Sq 


F value 


Trt 


I 


167.0 


167.0 


5. 1714 


ASN 


I 


32.0 


32.0 


0.9909 


WHT 


I 


142.7 


142.7 


4.4184 


SED 


I 


968.5 


968.5 


29.9864 


GAT 


I 


20.2 


20.2 


0.6260 


ERE 


I 


27.5 


27.5 


0.8525 


TIM 


I 


175.5 


175.5 


5.4341 


TIE 


I 


15.2 


15.2 


0.4696 


PreTest 


I 


14783.3 


14783.3 


457.7076 


ASN:TIM 


I 


271.7 


271.7 


8.4120 


SED:TIE 


I 


149.5 


149.5 


4.6272 


Trt:TIE 


I 


138.2 


138.2 


4.2800 


Trt:SED 


I 


II6.3 


II6.3 


3.6015 



SREE Spring 2012 Conference Abstract Template 



B-2 




